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Abstract 

Successful integration of UAS in the NAS will 
require that UAS interactions with the air traffic 
management system be similar to interactions 
between manned aircraft and air traffic management. 
For example, UAS response times to ATCo 
clearances should be equivalent to those that are 
currently found to be acceptable with manned 
aircraft. Prior studies have examined communication 
delays with manned aircraft. Unfortunately, there is 
no analogous body of research for UAS. The goal of 
the present study was to determine how UAS pilot 
communication and execution delays affect ATCos’ 
acceptability ratings of UAS pilot responses when the 
UAS is operating in the NAS. Eight radar-certified 
controllers managed traffic in a modified ZLA sector 
with one UAS flying in it. In separate scenarios, the 
UAS pilot verbal communication and execution 
delays were either short (1.5 s) or long (5 s) and 
either constant or variable. The ATCo acceptability 
of UAS pilot communication and execution delays 
were measured subjectively via post trial ratings. 
UAS verbal pilot communication delay, were rated as 
acceptable 92% of the time when the delay was short. 
This acceptability level decreased to 64% when the 
delay was long. UAS pilot execution delay had less 
of an influence on ATCo acceptability ratings in the 
present stimulation. Implications of these findings 
for UAS in the NAS integration are discussed. 

Introduction 

The FAA Modernization and Reform Act of 
2012 calls for a plan to integrate unmanned aircraft 
systems (UAS) into the National Airspace System 
(NAS) by 2015 [1], To achieve the goal of safe and 
non-disruptive flight in the NAS, however, UAS may 


be required to meet the same operational and 
certification standards as manned aircraft; they may 
have to “act and respond as manned aircraft do” 
(ICAO, 2011, p. 5) [2], Research is required to 
precisely quantify what these standards should be [3]. 
Flowever, although FAA regulations require that 
pilots respond promptly to air traffic controller 
(ATCo) commands and communications, they do not 
specify what an acceptable response delay actually is. 
The only stipulation is that delays must not 
compromise the safe separation of aircraft in the 
surrounding airspace. The present study examined 
how delays in verbal communications and command 
executions affected ATCo acceptability ratings of 
UAS and manned aircraft operating in a simulated 
NAS environment. 

Currently, ATCos direct traffic by issuing verbal 
clearances to aircraft. Pilots read back the 

clearances, and maneuver their aircraft to carry out 
the instructions. ATCos are required to monitor pilot 
read backs, and to correct pilots if they make a 
mistake [4]. The end-to-end response time to ATCo 
clearances is referred to as the “measured response” 
(MR) [5,6]. This overall response time can be broken 
down into the following MR components: (1) Pilot 
verbal communication delay. The lag between the 
end of an ATCo clearance and the beginning of the 
pilot’s readback. (2) Pilot execution delay. The lag 
between the end of the ATCo’s command and when 
the pilot begins to initiate the maneuver (3) Aircraft 
response delay. The time between the pilot entering 
a command and the aircraft initiating action. (4) 
ATCo display delay. The time for the maneuver to be 
visible on the ATCo display once the aircraft has 
begun its maneuver. In the present study, we focus on 
the first two MR components — pilot verbal 
communication delay and pilot execution delay. We 
did not examine aircraft response time delays or 


display visibility delays because these cannot be 
manipulated by the version of the Multiple UAS- 
Simulator (MUS1M) ground control station (GCS) 
that we used for the study [7]. Specifically, the 
version of MUS1M used in the present simulation 
was not programmed to simulate a specific UAS; the 
aircraft responded immediately once pilots entered 
commands. Moreover, UAS maneuvers were visible 
on ATCo radar screens within a constant time period. 

UAS differ from manned aircraft in several ways 
that are likely to affect their measured response. 
These include their size, speed, interfaces, and most 
critically, that UAS pilots are not co-located with 
their aircraft [8]. This fact can delay the response of 
both the pilot and UAS to ATCo commands, and it is 
important to determine what acceptable delays are. 

Research has been conducted on some of the 
MR components for manned aircraft. Cardosi [9], for 
example, examined verbal exchanges between pilots 
and controllers from three en route sectors with the 
goal of identifying the time from the ATCo’s initial 
transmission to the end of the pilot’s response, and 
assessing the various components therein. She found 
a mean total communication time of approximately 
10 sec, across three different types of clearances. 
Within this overall time, she found a mean pilot 
verbal delay of 3.31 sec (SD = 4.80 sec) for 
maneuvers of traffic avoidance, a mean verbal delay 
of 2.68 sec (SD = 4.60 sec) for turns not for traffic, 
and a mean verbal delay of 2.67 sec (SD = 6.25 sec) 
for traffic advisories. Based on these results, 
however, one cannot determine what an acceptable 
pilot delay is. This is because of the large variability 
associated with each of these means, and the fact that 
the means are based on some pilot delays that were 
too long to be acceptable: On 12% of the 
communications, controllers had to repeat their 
commands and some of these were because the pilots 
initially failed to respond. 

In a study conducted with recordings from 
TRACONs, Smith [10] found communication times 
between ATCos and pilots between 5.18 sec and 8.62 
sec depending on the type of maneuver. The delays 
between pilot and controller responses were also 
shorter than those identified by Cardosi [9], at about 
1 sec. However, the latter includes both pilot verbal 
delays (i.e., when the ATCo initiates communication) 
and ATCo verbal delays (i.e., when the pilot initiates 


communication), making it difficult to determine 
ATCo acceptability of this range. 

To summarize, based on the research on manned 
aircraft, it is difficult to quantify the acceptability of 
pilot delays in verbal responses. There is also little 
known about how long it takes pilots to begin 
executing commands and what is acceptable in this 
regard. We have begun conducting research to 
examine the acceptability of different MR 
components. Shively et al. [5], for example, had two 
ATCos issue 15 different clearances to pilots flying 
UAS using the MUSIM simulator. ATCos were not 
managing other traffic, and the pilots were not 
engaged in other UAS tasks. The pilots had to repeat 
the clearances, and execute the commands as quickly 
as possible. The different components of the MR 
were recorded, and the controllers rated the 
acceptability of the delays. The results revealed an 
average pilot communication delay of 2.5 sec, which 
is likely shorter than it would be under less ideal 
circumstances where pilots are engaged in other UAS 
tasks and are not expecting ATCo commands. 
Furthermore, the execution delay was found to vary 
considerably, ranging from 1.7 sec to 7.63 sec. For 
some clearances, execution delays were found to be 
negatively correlated with ATCo acceptability 
ratings. That is, longer execution times were 
associated with lower acceptability ratings. 

The present study furthers this research by 
manipulating the first two MR components and 
examining the consequences for ATCo acceptability 
ratings. In particular, we manipulated whether UAS 
pilot verbal responses were “short” or “long” by 
inserting a delay of either 1.5 sec or 5 sec to their 
verbal responses. We also manipulated whether the 
execution delays were short or long by adding a delay 
of either 1.5 sec or 5 sec prior to the beginning of 
their responses. The values for the pilot verbal delay 
were chosen based on the acceptability ratings 
collected in Shively et al. [5], where pilot verbal 
delays greater than 5 sec were generally rated 
unacceptable by the ATCos. The current study also 
manipulated whether the delays were constant or 
variable. In the constant conditions, the scenarios 
either had all short or all long verbal delays, paired 
with either short or all long execution delays. In the 
variable conditions, however, some verbal delays 
were long and some short, and the same was the case 
with the execution delays. This allowed us to 
determine whether it is a delay per se that is 


problematic, or whether it is the predictability of the 
delays that is essential to their acceptability. It is 
possible, for example, that if the delays are constant, 
ATCos can develop strategies for accommodating 
them. This would be more difficult if the delays are 
unpredictable. 

Method 

Participants 

Participants were 8 radar-certified Air Traffic 
Controllers, with a mean of 28 years of experience in 
military and civilian air traffic management. Six of 
the participants were recently retired. Participants 
were compensated $60 per hour for their time. 

Design 

This simulation followed a 2 (Verbal Delay: 
Short or Long) x 2 (Execution Delay: Short or Long) 
x 2 (Predictability Constant or Variable) within- 
subjects design. The additional delays were 1.5 
seconds for the Short condition and 5 seconds for the 
Long condition. This generated four combinations of 
voice-execution delays: Voice-Short Execution- 

Short, Voice-Short Execution-Long, Voice-Long 
Execution-Short, and Voice-Long Execution- 
Long. The Constant delay condition used only one of 
the four delay combinations through an entire trial. 
The Varied delay condition switched between delay 
combinations during the trial to include two 
repetitions of each delay combination in random 
order. 

Apparatus 

The simulation was conducted in the Center for 
Human factors in Advanced Aeronautics 
Technologies (CHAAT) at California State 
University, Long Beach. The simulation was run in 
the Multi Aircraft Control System (MACS) [11], 
which creates a medium-fidelity airspace 
environment. MACS simulated a DSR station for 
sector ZLA 20. All manned aircraft were flown by 
one pseudopilot using MACS in pseudopilot mode; 
the UAS was flown by a pseudopilot on the MUSIM 
GCS. In MUSIM, the pseudopilot controlled the 
altitude and point-to-point navigation of the UAS. 
The UAS (callsign PD-1) flew flight paths at 


altitudes between 10,000-16,000 ft. and at a constant 
cruise speed of 1 10 knots. 

The ATCo, manned pseudopilot, and UAS 
pseudopilot communicated via push-to-talk headsets 
over a voice server. On MUSIM, voice transmissions 
were delayed by either 1.5 s or 5 s, depending on the 
delay condition. The voice communication system 
also allowed for the production of “step-ons” in the 
event of simultaneous transmissions. 

Procedure 

The simulation was conducted over two days for 
each participant. On the morning of the first day 
participants completed consent forms, demographic 
questionnaires, and were briefed on simulation 
procedures, following the briefing the controllers 
performed four 30-minute practice trials. Then, the 
ATCos performed two experimental blocks, each 
consisting of four, 40-minutes scenarios (one block in 
the afternoon of Day 1 and one block in the morning 
of Day 2). During the experimental trials, controllers 
managed all air traffic coming into their sector, which 
consisted of three LAX arrival streams and 
overflights. Controllers were instructed that arrival 
aircraft had priority and were required to leave the 
sector at an altitude no greater than 12,000 feet, and 
airspeed 250 knots. The controllers were told that 
their Center had a Letter of Agreement to 
accommodate PD-l’s flight objectives while 
maintaining safe operations for all air traffic. 

Beginning approximately 1-2 minutes from the 
start of the trial and every 4-5 minutes in alternating 
order, either "Mission Control" or the UAS would 
initiate a route modification request for PD-1. 
Mission Control was simulated on a separate 
computer located to the right of the controller’s radar 
scope. This station would alert the controller through 
his headset and display a heading modification (for 
the UAS (e.g., "PD-1 proceed direct EDITS"). 
Mission control only requested the ATCo to direct 
the UAS to fixes. The ATCo had to issue PD-1 a 
new clearance to the fix. The UAS would request 
altitude clearances while flying to a directed fix. A 
total of 8 UAS changes were completed in each 
scenario, four initiated by the controller and four 
initiated by the UAS pseudopilot. 

Voice and Execution delays were controlled at 
the UAS pseudopilot station. Voice software at this 


station automatically delayed transmission of the 
UAS pseudopilot’s audio for the set 1.5 or 5 seconds 
before broadcasting it to the ATCo and manned 
pseudopilot. The UAS station only included a 
transmission delay, not a receiving delay. The ATCo 
and manned pseudopilots had no added receiving or 
transmission delays. After verbally responding to the 
clearance from the ATCo, the UAS pilot would 
activate a countdown timer of 1.5 or 5 s before 
beginning to execute the instruction. For the variable 
delay conditions, the voice and execution delays 
changed between clearances. Each voice-execution 
delay combination was presented twice in random 
order. 

After each trial, controllers rated their situation 
awareness, workload, experience, and acceptability 
of interacting with the manned and UAS 
pseudopilots. Once all 8 experimental trials were 
completed, controllers answered post-simulation and 


debriefing questionnaires. Finally, the ATCos were 
interviewed during a debriefing session. 

Results 

The data reported in this paper represents only a 
subset of the data collected from the simulation. In 
the present paper, we report descriptive analyses of 
the two MR components and the acceptability rating 
data provided by ATCos for these components. Data 
from one participant was excluded from the analyses 
due to noncompliance with the experimental 
instructions. 

The means and standard deviations for MR 
components 1 and 2, and their corresponding ATCo 
acceptability ratings are listed in Tables 1 and 2, 
respectively. Because the ATCo acceptability ratings 
were collected post-trial, the mean ratings are 
collapsed across the four variable scenarios. 


Table 1. Means and Standard Deviations for the MR Component 1, Verbal Response (s) and 

Acceptability Ratings. 

Condition Pilot Role MR Component 1 Mean ATCo 

Communication Lag Acceptability Rating 




Mean 

Std. Dev 

Mean 

Std. Dev 

Constant: VS-ES 

UAS 

2.07 

0.50 

5.14 

0.69 


Manned* 

0.81 

0.23 

5.00 

1.41 

Constant: VS-EL 

UAS 

2.12 

0.27 

4.86 

1.57 


Manned* 

0.80 

0.35 

5.43 

1.13 

Constant: VL-ES 

UAS 

5.52 

0.32 

4.00 

1.41 


Manned* 

0.86 

0.34 

4.43 

1.51 

Constant: VL-EE 

UAS 

5.43 

0.25 

4.43 

1.39 


Manned* 

1.15 

0.51 

4.43 

1.13 

Variable 

(averaged across the 
4 variable scenarios) 

UAS 

3.72 

0.22 

5.07 

1.17 

Manned* 

0.94 

0.52 

5.43 

1.02 


*For manned aircraft pseudopilots, only a small sample of pilot-ATCo interactions were examined 


Table 2. Means and Standard Deviations for the MR Component 2, Execution Response (s) and 

Acceptability Ratings. 


Condition 

Pilot Role 

MR Component 2 
Execution Lag 
Mean Std. Dev 

Mean ATCo 
Acceptability Rating 
Mean Std. Dev 

Constant: VS-ES 

UAS 

6.27 

2.51 

5.14 

0.69 


Manned* 

3.24 

2.21 

5.00 

1.83 

Constant: VS-EL 

UAS 

10.57 

1.32 

5.00 

1.00 


Manned* 

4.79 

1.47 

5.43 

0.98 

Constant: VL-ES 

UAS 

7.17 

1.43 

4.43 

1.90 


Manned* 

5.56 

3.48 

5.29 

0.95 

Constant: VL-EL 

UAS 

9.98 

2.34 

4.43 

1.13 


Manned* 

4.97 

2.45 

5.29 

1.38 

Variable 

(averaged across the 
4 variable scenarios) 

UAS 

8.98 

1.24 

5.18 

1.00 

Manned* 

5.58 

2.43 

5.71 

0.78 


*For manned aircraft pseudopilots, only a small sample of pilot-ATCo interactions were examined 


MR 1: Pilot Verbal Communication Delay 

Mean MR-1 component times were computed 
for each participant in each condition. For the 
variable condition, we extracted the data for the 
specific voice and execution delay implemented in 
the pilot-ATCo interaction. Mean response time for 
MR1 was submitted to a 2 (Verbal delay: short vs. 
long) x 2 (Execution delay: short vs. long) x 2 
(Predictability: constant vs. variable) x 2 (Pilot Role: 
UAS vs. manned) mixed ANOVA. Pilot Role was 
the only between-subjects variable. 

The main effect for Verbal Delay was 
significant, F’(l,12) = 2004, p < .001. As expected, 
the time interval between the end of an ATCo verbal 
clearance and the pilot’s read back was shorter in the 
conditions with the short UAS verbal delays ( M = 
1.41 sec) compared with long delays (M = 3.27 sec). 
In addition, there was a significant main effect of 
Pilot Role, F{ 1,12) = 959, p < .001, where the verbal 
communication lag was shorter for manned aircraft 
(M = 0.92 sec) than the UAS ( M= 3.75 sec). Again, 


this finding is expected because only the UAS 
transmissions were delayed. These two main effects 
were qualified by a significant two-way interaction of 
both variables, F(l,12) = 1502, p < .001. As shown 
in Figure 1, there was a large difference in UAS 
verbal communication delay as a function of the 
verbal delays that was not evident with manned 
aircraft. 
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Figure 1. Verbal Response Time (MR-1) as a 
Function of UAS Verbal Delay x Pilot Role 


MR 2: Pilot Execution Delay 

The same ANOVA was run for the MR-2 
component. The main effects for Verbal Delay, 
F( 1,12) = 5.5, p = .03, and Pilot Role, F(l,12) = 23.4, 
p < .001, were significant. The time from when the 
ATCo completed issuing the clearance to when the 
pilot started to execute the clearance was shorter in 
the conditions with the short UAS verbal delay (M = 
6.69 sec) than long delay (M= 7.16 sec). In addition, 
the execution time was shorter for manned aircraft 
(M= 5.1 1 sec) than the UAS (M= 8.74 sec). 

For MR 2, the main effects of Execution Delay, 
F(l,12) = 56.3 , p < .001, and Predictability, F(l,12) = 
4.7, p = .05, were also significant. Pilots initiated the 
execution of a clearance faster for short UAS- 
execution delays (M= 6.21 sec) than long delays (M 
= 7.64 sec). The execution response was also faster 
when the delays were constant (M = 6.57 sec) than 
variable (M= 7.64 sec). 
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Figure 2. Execution Response Time (MR 2) as a 
function of UAS Execution Delay x Pilot Role 
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Figure 3. Execution Response Time (MR 2) as a 
function of UAS Execution Delay and 
Predictability 


Execution delay entered into two, two-way 
interactions; one with Pilot Role, F(l, 12) = 83.6, p < 
.001, and the other with Predictability, A(l,12) = 6.2, 
p = .03. As shown in Figure 2, there was a large 
difference in UAS pilots’ MR 2 response times as a 
function of the induced pilot execution delays that 
was not evident with pilots of manned aircraft. 
Figure 3 illustrates that the difference between the 
short and long execution delay was more evident 
when the delay is constant than when it is variable. 

ATCo Acceptability Ratings 

Because the ATCo acceptability ratings were 
collected post-trial, we could only examine the 
effects of the verbal and execution delay variables 
independently for the constant delay conditions. 
Acceptability was rated on a 1 (not at all acceptable) 
to 7 (very acceptable) scale. 

Because the ATCos rated both the acceptability 
of pseudopilots flying UAS and manned aircraft, a 2 
(Verbal delay: short vs. long) x 2 (Execution delay: 
short vs. long) x 2 (Pilot Type: UAS vs. 

conventional) repeated measures ANOVA was 
performed. No effects were significant. In general, 
the overall communication lag and execution lag in 
all conditions were judged to be acceptable (all 
means greater than or equal to the midpoint, i.e., 4.0, 
of the acceptability rating scale). To examine the 
effects of predictability, a second ANOVA was 
performed on acceptability ratings with Predictability 
(constant vs. variable) as a factor. Again, this 
analysis yielded no significant effects. 

Correlations were computed between ATCo 
acceptability rating and MR component for the 
constant verbal and execution delay conditions 
separately for UAS and manned aircraft. For UAS, 
the correlation between acceptability ratings and MR 
1 was marginally significant, r = -0.35, p = .07 but 
the correlation between acceptability and MR 2 was 
nonsignificant, r = -0.19. Shorter verbal delays were 
associated with higher acceptability ratings. For 
manned aircraft, the correlations were nonsignificant 
for MR 1, r = -0.29 and MR 2, r = -0.03. Note also 
that acceptability ratings of MR 1 and MR 2 were 
significantly correlated, r = 0.74, p < .001, suggesting 
that ATCos had difficulty separating the contribution 
of each measured response to acceptability. 
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Figure 4. Distribution of UAS MR-1 Acceptability 
Ratings for Short and Long Communication 
Delays 

Figure 4 shows the distribution of acceptability 
ratings of short and long delays for UAS. For short 
delays, 92% of the ratings were acceptable; for long 
delays, 64% of the ratings were acceptable. Figure 5 
shows these ratings for manned aircraft. Again, the 
majority of the ratings were in the acceptable range: 
92% of the ratings were acceptable for short UAS- 
communication delays, and 71% were acceptable for 
long UAS-communication delays. 
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Figure 5. Distribution of Manned Aircraft MR-1 
Acceptability Ratings for Short and Long UAS 
Communication Delays 

Discussion 

The present study examined the effects of UAS 
verbal and execution delays on ATCo acceptability 
ratings of a UAS operating in the NAS. We were 


able to extract the Pilot verbal communication 
response latency (MR 1 component) and execution 
response latency (MR component 2) in the simulation 
for UAS and manned aircraft. 

For the pilot verbal communication delays, the 
mean response latency for manned aircraft was 0.78 s 
for the short delay condition and 1.06 s for the long 
delay condition. Thus, the induced UAS pilot verbal 
communication delay also lengthened the 
communication responses of manned aircraft. This 
lengthening in communication lag for the manned 
aircraft could be a result of step-ons that were 
associated with the long UAS verbal delays, because 
the number of times the UAS transmissions stepped 
on the transmissions of ATCos or manned aircraft 
increased in the long verbal-delay conditions. 

Thus, inclusion of the UAS with long pilot 
verbal communication delays can push the manned 
aircraft pilots’ verbal communication lag into 
unacceptable regions, even though the latencies for 
manned aircraft were 3-5 times faster than UAS 
latencies. Nevertheless, in the present simulation, the 
responses of manned aircraft were rated unacceptable 
on 29% of the long delay simulation trials, and were 
rated unacceptable on 8% of the short delay 
simulation trials. For long delays the percentage of 
unacceptable ratings is larger than the 12% of 
communications that ATCos had to repeat in the 
study conducted by Cardosi [9] . 

The UAS pilots’ verbal communication delay 
was 2.03 s in the short verbal delay condition and 
5.53 in the long verbal delay condition. 
Consequently, only 64% of the acceptability ratings 
were in the acceptable range for long delays. 
Flowever, the UAS execution delay was not related to 
ATCo acceptability ratings for either manned aircraft 
or UAS. One possible reason for this lack of an 
effect of execution delay was the slow speed (PD-l’s 
speed was only 110 knots) of the UAS. During the 
debriefing session, the ATCos indicated that they did 
not really pay attention to the execution time because 
the slow speed of UAS meant that the execution 
delay did not disrupt other traffic. The ATCos 
indicated that the execution delay would have been a 
bigger factor if the UAS was faster. For manned 
aircraft, all but one execution time was rated as 
acceptable (4 or higher) with the outlier given a 
rating of 3. As such, there was little variability in 


acceptability ratings for the execution times for the 
conventional pilots. 

In conclusion, our simulation showed that an 
average UAS pilot verbal communication delay of 2 
sec was acceptable nearly 100% of the time, but an 
average delay of 5.5 sec was only acceptable slightly 
more than half of the time. Note also ATCos rated a 
manned aircraft delay as unacceptable on about 30% 
of the transmissions in the long delay condition, 
despite the fact that this value is much less than the 
verbal delay of the UAS. The additional tolerance in 
pilot verbal delay shown by controllers is likely a 
result of the ATCos’ accommodations for the one 
UAS in their sector. Moreover, the fact that 
acceptability ratings of communication and execution 
delays were highly correlated suggests that ATCos 
may have had difficulty determining the unique 
contribution of each MR component to acceptability. 
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